Modern cyber security operations collect an enormous amount of logging and alerting data. While analysts have the ability to query and compute simple statistics and plots from their data, current analytical tools are too simple to admit deep understanding. To detect advanced and novel attacks, analysts turn to manual investigations. While commonplace, current investigations are time-consuming, intuition-based, and proving insufficient. Our hypothesis is that arming the analyst with easy-to-use data science tools will increase their work efficiency, provide them with the ability to resolve hypotheses with scientific inquiry of their data, and support their decisions with evidence over intuition. To this end, we present our work to build IDEAS (Interactive Data Exploration and Analysis System). We present three real-world use-cases that drive the system design from the algorithmic capabilities to the user interface. Finally, a modular and scalable software architecture is discussed along with plans for our pilot deployment with a security operation command.

Microsoft recently released two new data science tools for interactive data exploration: modeling and reporting. These data science utilities, called Interactive Data Exploration, Analysis and Reporting (IDEAR) and Automated Modeling and Reporting (AMAR) can be reused by data science teams for specific tasks in their projects. Data science teams spend a significant amount of time writing code to answer data related questions like data schema, missing data elements, individual variable distribution & transformation, specific clustering patterns in the data, and the performance of Machine Learning (ML) models. These two tools can be used to automate these common tasks in the data science lifecycle. The goal is to ensure consistency and completeness of data science tasks across different projects in the organization.